Decoding the Hype

Identifying real evidence in health & nutrition studies

John (The John & Calvin Podcast)

Health and nutrition research is messy.

  • No study is flawless. Every design has limits.
  • Headlines & influencers turn nuance into clickbait.
  • The Fix:
    study design • basics stats • common pitfalls

Overview

  1. The Research Question
  2. Study Population
  3. Study Design (Power)
  4. Study Structure
  5. How Results Are Reported
  6. Interpreting Results
  7. Controlling for Other Factors
  8. Misleading Studies & Influencers

The Research Question

Main goal of the study

  • Drives study design, statistics, sample size
  • Includes measured metric (e.g., BP change, risk)
  • States the primary outcome (vs. exploratory or secondary)

Best practice (especially in RCTs): pre-register outcomes and analysis plan

Example (published RCT)
“We tested the hypothesis that long-term supplementation with omega-3 fatty acids would reduce cardiovascular events in this population.”
NEJM, 2012

The Research Question

Questions to Ask

  • What single question is the study trying to answer?
  • Is the primary outcome clearly stated and the right one for that question?
  • Are secondary outcomes labelled as exploratory, or being oversold?
  • Does this question matter to you?

Study Population

Population matters for relevance
Not all studies are done on humans, and not all humans are like you.

  • Cells (in vitro)
  • Animals (in vivo)
  • Humans (in vivo)

Study Population

Questions to Ask

  • Who or what was actually studied?
  • Why did the researchers pick that population or model?
  • How similar are these subjects to you or the group you care about?

Study Design

What a study can claim depends on its design.


Observational

• Cross-sectional
• Prospective cohort
• Case-control

Interventional

• RCT
• Pre–post (single-arm)

Evidence Synthesis

• Systematic review
• Meta-analysis

Other Study Designs (reference)


Observational

• Case-control – with vs without outcome
• Case report / series – detailed look at few patients
• Ecological – group-level data only
• Retrospective cohort – past records follow exposure → outcome


Mixed / Natural

• Longitudinal – same subjects over time
• Natural experiment – exposure assigned by external factors

Interventional

• N-of-1 trial – single participant, alternating treatments
• Cross-over trial – each subject receives all treatments
• Before–after study – compare pre- vs post-intervention


Synthesis

• Systematic review – structured summary, no pooling
• Umbrella review – review of systematic reviews

Study Design

Sample Size & Power

  • Power (1 − β): the probability of finding an effect if it really exists
    Usual target: 80%

  • Grows with bigger N, larger effect, lower noise

  • Calculated before the trial to size it properly
  • Set from the minimum clinically important difference (MCID) (ideally)

Study Design

Sample Size & Power

  • Too small → wide CIs, missed effects
  • Too large → trivial but ‘significant’ findings
  • Rule-of-thumb study sizes
    • < 50 per arm = pilot / feasibility
    • 50 – 300 per arm = typical nutrition RCT
    • > 300 per arm = large, clinically robust
  • Power targets the primary outcome; secondaries often under-powered

Study Design

Questions to Ask

  • What type of study is this: observational, interventional, or a synthesis?

  • Can this design support a causal claim, or only association?

  • Is the control / comparison group appropriate?

  • For meta-analyses, are the pooled studies similar enough?

  • Was the sample size justified, and did they report a power calculation for the primary endpoint?

Study Structure

  1. Abstract
  2. Introduction
  3. Methods
  4. Results
  5. Discussion
  6. Conclusion
  7. Supplementary

Study Structure

Questions to Ask

  • Does the Abstract match what’s really in the paper?
  • Are the Methods detailed enough to judge rigor?
  • Are Results shown with raw numbers, effect sizes and confidence intervals?
  • Are any Results missing or highlighted more than reasonably?
  • Do the Discussion/Conclusion stay within the data’s limits?

How Results Are Reported

Describing the Population

  • Counts – raw number (e.g. 500 cases)
  • Percent / Proportion – share of population (e.g. 5%)
  • Incidence – new cases in a time period (e.g. 100 new this year)
  • Prevalence – all existing cases at a point in time (e.g. 5% of adults)
  • Rates – events per person-time (e.g. 10 per 1,000 person-years)
  • Standardized rates – adjusted for age or other factors to compare groups

How Results Are Reported

Measuring Differences in Outcomes

  • Mean difference – raw difference (e.g. −5 mm Hg in blood pressure)
  • Median difference – comparison of middle values (less sensitive to outliers)
  • Percent change – relative difference from baseline or control
    (e.g. treatment group: 180, control group: 200 → 10% lower in treatment)

How Results Are Reported

Understanding and Comparing Risk

Describing Risk in One Group

  • Absolute risk – chance of outcome in one group (e.g. 1%)
  • Odds – ratio of events to non-events (e.g. 1:9 odds = 10% chance)

Comparing Risk Between Groups

  • Absolute Risk Reduction (ARR) – difference in risk (e.g. 2% → 1% = 1% ARR)
  • Relative Risk (RR) – ratio of risk (e.g. 1% ÷ 2% = RR 0.5 = 50% lower risk)
  • Odds Ratio (OR) – ratio of odds (e.g. OR 2 = twice the odds)
  • Number Needed to Treat (NNT) – how many need treatment for one to benefit (e.g. NNT = 100)

How Results Are Reported

Tracking Outcomes Over Time

  • Incidence rate – new cases per person-time (e.g. 5 per 1,000 person-years)
  • Hazard ratio (HR) – event rate over time (e.g. HR 0.75 = 25% lower risk)
  • Survival curves
    probability of staying event-free over time
    (e.g. Kaplan–Meier plot)

Kaplan–Meier analysis on the association of obesity and malnutrition with the risk of all-cause death in hypertension patients.

How Results Are Reported

Summarizing Multiple Studies

  • Standardized Mean Difference (SMD) – compares effects when scales differ
  • Forest plot – visual summary of study results and overall effect

Forest plots of saturated fatty acid reduction trials on myocardial infarction.

No statistically significant reduction was noted.

How Results Are Reported

Questions to Ask

  • Which metric is used: count, rate, risk ratio, mean difference, etc.?
  • Is that metric appropriate, or could it mislead (e.g., relative vs. absolute)?
  • Does it answer the main research question?
  • Could the metric exaggerate or downplay the finding?

Interpreting Results

P-values & Confidence Intervals


Is this effect real? How precise is it?


Concept What it tells us Quick example
Statistical significance (p-value) Chance vs. real effect? Supplement ↓ BP 5 mm Hg, p = 0.03
Precision (95 % CI) How exact is the estimate? 5 mm Hg (CI −8 to −2)

Interpreting Results

The p-value

  • Threshold test: typically p < 0.05
  • Smaller p means stronger evidence against chance
  • A p-value means little unless you also know the effect size
  • p ≠ probability the result is true

“5 mm Hg decrease, p = 0.03” → significant

Interpreting Results

Statistical vs. Practical Significance


Finding (effect) p-value Interpretation
−0.15 kg (12 wk) 0.001 Statistically significant; clinically trivial
−8.5 mm Hg systolic BP 0.09 Not statistically significant; could matter if real


Interpreting Results

Questions to Ask

  • Is the result statistically significant and by how much?
  • What does the confidence interval say about size and precision?
  • Is the effect big enough to matter in real life?
  • If it’s borderline, could high variation, small N, selective data processing or multiple testing explain it?

Core Questions to Ask

  1. Is the main question clear, and is it the one you care about?
  1. Study type: does it address causation, or only association?
  1. Who / what (and how many!) was studied, and how relevant is that to you?
  1. Metric chosen: what does it reveal and what does it hide?
  1. Result strength: significant? How large, precise, and practically useful?
  1. Red flags: statistical ‘tricks’, sensational claims, or conflicts?

Studies Can Mislead

  • Small samples inflate effects
  • Placebo isn’t inert
  • Publication bias
  • Funding or pet theories
  • Multiple comparisons (p-hacking), cherry-picking subgroups
  • Self-reported or recall data
  • Lack of blinding
  • Surrogate ≠ clinical outcome

Influencers Do Mislead

  • One study ≠ truth
  • Petri-dish (mechanism) hype
  • Healthy Diet ≠ morality
  • Relative risk without absolute numbers
  • Association ≠ causation
  • Anecdotes ≠ strong evidence
  • Claim comes from headline, not published study

Three Things to Remember

  1. Know the study’s main question
    It’s the foundation for understanding everything that follows.

    Don’t rely on the headline / influencer.


  1. Understand the study design
    It defines what the results can tell you and how they should be interpreted.

    Take into account the full body of evidence.


  1. Don’t skip the statistics
    Results are only as strong as the methods and statistical analysis behind them.

    Understanding the basics goes a long way.

Follow-up on Vegan Twin Study